big discrepancy
[D] Is overfitting a problem if validation R2 is maximized?
I am trying to figure out how to deal with overfitting in my data. Basically no matter what hyper-parameters I tweak, I see a big discrepancy between my validation-set R2 and my in-fit R2 (r2 0.8 for in fit, 0.25 for validation). Typically this is a seen as a "red-flag" for overfitting, but there is no way for me to close the gap between in- and out-of-fit R2 without lowering my overall validation R2. Below is the plot of validation R2 vs in-fit R2 for 15,000 different hyper-parameter combinations. The only way I can have maximum out-of-fit accuracy is by having a big discrepancy between in and out of fit R2.